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The purpose of this paper is to look for links in a virtual trainee's interest and self-efficacy in a simulated 
event as it relates to their previous self-reported technical skill level. Ultimately, the idea would be to 
provide the right amount of support at the right place at the right time to set the conditions for maximum 
transfer of the skill sets to the work place. An anecdotal recap of a recent experiment of a medium-scale 
training event produced in a virtual world will provide examples for discussion. In July 2010, a virtual 
training event was produced for the Air Force Research Lab's Games for Team Training (GaMeTT) at the 
Patriot Exercise at Volk Field in Wisconsin. There were 29 EMEDS participants who completed the 
simulated OCO event using the OLIVE gaming engine. Approximately 25 avatars were present at any 
given time; including role players, observers, coordinators and participants. 


1.0 INTRODUCTION 

There is a growing number of high fidelity, 
well-developed, multi-player gaming 
engines available to organizations for the 
purpose of training individuals and 
geographically dispersed teams. These 
gaming engines provide a variety of options 
and capabilities, from which detailed 
simulated events can be conceptualized 
and executed virtually. This trend toward 
virtual world training opens up a plethora of 
options for organizations to accomplish 
learning objectives and design experiences 
that allow team functioning and practice in a 
safe environment. 

Occasionally these objectives can become 
overwhelming to participants with little or no 
previous gaming experience. At best, 
individual, experienced gamers can interact 
in the virtual world with little interruption, but 
participant teams are seldom 
homogeneously technophiles. The task of 
supporting the accomplishment of training 
objectives executed in a virtual environment 
with participants of wildly varying technical 
skill sets can become a barrier to the 
achievement of the objectives. 


With this idea in mind, participants of an 
experiment with a simulated event in a 
virtual world were recently asked questions 
related to their technical skill level, interest 
level in virtual training and self-efficacy. 

The purpose of this paper is to report upon 
the responses and look for links in a virtual 
trainee’s interest and self-efficacy in a 
simulated event as it relates to their 
previous self-reported technical skill level. 
Ultimately, the idea would be to provide the 
right amount of support at the right place at 
the right time to set the conditions for 
maximum transfer of the skill sets to the 
work place. An anecdotal recap of a recent 
experiment of a medium-scale training 
event produced in a virtual world will provide 
examples for discussion. . 

2.0 BODY 

The experiment took place at the close of 
the above mentioned Patriot Exercise on 
July 20th and 21st at building 533 at Volk 
Field near Tomah Washington. This 
particular building is on the flight line of the 
field and consists of several large rooms 
connected by halls with smaller office 
spaces. The rooms were equipped with 
ample seating, though minimal desk space 
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for computer set up. The building contained 
a SIPRINET network which was unavailable 
for this unclassified experiment. However, 
the sponsors of the event were able to 
secure 2 DSL connections which were 
temporarily routed to the building and 
became the uplink for the Ethernet network 
engineers stood up for the event. 

Since wireless connections were not 
allowed, a router was connected to the DSL 
lines and Ethernet connections were 
established for each of 13 participant 
laptops used for participant and exercise 
support personnel. Two basic 

configurations were used during the 
experiment. The first configuration was a 
simple semi circle, with a trainer station in 
the center and a screen behind the trainer 
stand. 



The second configuration was needed in 
order to simulate a geographical distance 
between players, as would be the case in 
ultimate use of the GaMeTT Training 
System. So, after some initial training was 
completed, participants were scattered 
throughout the facility in various rooms and 
offices. To make this happen, the network 
configuration above was broken down and 
moved to create various access points The 
new seating/network arrangement was 
designed to produce as much space as 
possible between participants, mimicking 
the way in which users would connect from 


work or home. See the diagram below for 
this configuration: 



2.1 Details of the Event 

There were three groups {Wednesday PM, 
Thursday AM and Thursday PM) brought in 
by van at the close of the second week of a 
two week live training event for forward 
medical teams. It is noteworthy that each of 
the 29 participants arrived in the building 
after almost two weeks of live exercises 
which continued 24 hours a day for 10 days. 
Each of the three groups was composed of 
individuals ready to play the following roles 
in the simulated event: 

Nurse (2) 

Doctor {1) 

Administrative Officer (1) 

Administrative Technician (1) 

Medical Technician (3) 

Each of these medical personnel was given 
a 30 page user guide with quick reference 
charts and a live one hour intensive 
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instructional session on the basics of 
operating their avatar and functionality of 
the virtual world. This training was 
conducted in the semi-circle shown above. 
Additionally, one support person for every 3 
participants was available to answer 
questions. 

Operation of the simulation required the 
movement of an avatar, movement of 
objects in-world and the use of a series of 
menus to access the medical treatment 
model for trauma cases that were a part of 
the simulation. 
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After the initial training event, the 
participants were separated into the second 
configuration in order to simulate a 
distributed environment. In this setup - a 
minimum 1 live support person was 
provided for every 4 participants. 

During the execution of the event, it must be 
noted that the server housing the virtual 
environment went down twice and the DSL 
Internet connection feeding the 
experimental network service was 
interrupted a minimum of three times. 


2.2 Evaluation Criteria 

The overall evaluation methodology was 
based on Gagne’s Nine Instructional 
Events, Gagne proposed that if learning 
content contained 9 significant elements, 
the optimum conditions for learning would 
be created for the transfer of learning from 
the training environment to the real world. 
The graphic below represents these 9 
events {Gagne, 1985). 

This approach to the assessment and 
ultimately evaluation of the assessment 
results takes into account the fact that the 
transfer of training knowledge, skills and 
abilities can be difficult to achieve and 
measure in any setting. Furthermore, it 
acknowledges that the use an inquiry-based 
approach provides a holistic and iterative 
developmental process for solving the 


Gagne’s hierarchical model builds from the 
lowest level (Gaining attention) and works 
its way up to “Enhanced retention and 
transfer” so that simulation architects can 
ensure that the optimum conditions for 
learning have been created. For example, it 
is important to build in environmental 
objects and training cues that get the 
learner's attention before attempting to 
share the learning objectives. Simulation 
architects, engineers, and instruction 
designers can use the methodology’s 
guidelines and checklists to assist with 
prioritization and to mark the distinction 
between desired from required elements. 
Tools such as checklists and guidelines can 
help streamline concurrent development 
occurring in three or four related but distinct 
design fields: engineering, instructional, 
graphical and logistics. (Gagne, 1977) 

The idea is to use a short cycle of feedback 
and assessment to influence the 
development/execution of the simulated 
event as opposed to an independent 
examination that is conducted only once at 
the dose of the project. Virtual world 
development is too intricate and multi- 
variable to be reduced to a single snap shot 
in time that produces and yes or no answer 

Meeting the requirements of the Gagne’s 
Nine Instructional Events can be more 
complex than it may seem because even 
though their effect is hierarchical, the 
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creation may be neither contiguous or 
chronological. 

The survey responses included in this paper 
are part of this nine layer approach to 
evaluating simulated events implemented in 
virtual worlds. In the Pre-Event Survey, the 
participants were asked to self-report their 
technical skill level directly as well as in a 
series of questions designed to give a 
relative sense of their technical abilities in 
relationship to others. Participants were 
also asked to rate their interest level in the 
concept of conducting training in a virtual 
context. 

In the Post- Event Survey, participants were 
asked if they felt they would apply their 
newly acquired skill sets on the job. 
Research shows that this concept of self- 
efficacy is a strong indicator to the transfer 
of skill sets from a training environment to 
the workplace. 

Learner characteristics that support transfer 
are self-efficacy, pre-training motivation and 
perceived utility. (Bandura, 1994) In fact, a 
number of recent studies indicate that self- 
efficacy is the primary indicator of whether 
or not participants will experience increased 
performance once returning to the work 
place. Further, the complex task of 
measuring performance improvement on 
the job and attributing that performance to 
causal factors related to training events can 
be short circuited by simply asking the 
participant how useful they found the 
learning. (Grille, 2000) Similarly, asking 
learners about their reasons for participating 
in a learning experience (are they motivated 
to learn) and whether or not the material is 
applicable to their work (will they use the 
skill on the job) is a positive indicator that 
transfer will occur. 

3.0 DISCUSSION 

Among all of the before mentioned details, 
there are several caveats. The first is that 
if the experiment were extended to include 
participants who are dispersed 
geographically, all training and support 


would be done using the web and/or VoIP 
connection. Training could possibly be 
conducted via webinar, which still means 
participants would need to ready their 
equipment with the appropriate downloads 
and hardware to run the virtual environment. 

This particular experiment was conducted 
live because the participants gathered for a 
live training exercise and were a “captive” 
audience on whom the tool could be tested. 

Results of the participant Pre-Survey show 
that 64% reported themselves as technically 
proficient and 69% appear to be tech-savvy 
to their peers. This similar number indicates 
some agreement among the direct and less 
direct questions regarding technical skill 
level. Also, there were 5 individuals who 
rated themselves the lowest possible 
number on the technically proficient scale 
and 4 individuals who rating themselves the 
highest possible number. 

When asked whether or not they were 
interested in learning more about virtual 
worlds, 13 individuals or 46% said they 
highly agreed, agreed, or were neutral. 
Likewise, 57% of the participants believed 
that virtual training can be effective for their 
team. Forty-six percent said they would be 
willing to train virtually when they return 
from the live exercise and 64% indicated 
they have high expectations for simulated 
events in virtual worlds. 

On the Post-Event Survey, a consistent 
64% said they would apply the skills learned 
in the simulated event to their work. 

4.0 CONCLUSION(S) 

These results are incredibly consistent and 
remarkably unremarkable considering the 
diversity of the group. The numbers may be 
interpreted to say that about the same 
number of highly tech-savvy individuals 
have high expectations and plan to apply 
their skills learned in the virtual world to the 
real world. These results span the Pre- and 
Post-Event surveys, therefore show little or 
no change in attitude. 
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There are two additional questions from the 
Post Event Survey that may reveal some 
interesting attitudes surrounding this 
simulated event in a virtual world. 

When asked “During the training session, I 
was provided with enough support to be 
able to adequately use the technology”, only 
29% of the participants highly agreed or 
agreed with the statement. Further, when 
asked to agree with the statement 
“Someone was available to answer my 
questions about the virtual world used in 
this training session.” a mere 5 individuals 
or 17% highly agreed or agreed. 

These low numbers beg the question - if 
the simulated event was executed, and 
participants were able to learn skill sets that 
would be applied to the work place, how is it 
that the users did not feel supported? 
Another interesting question would be, what 
types of support would be needed if 
participants were geographically dispersed 
and there was no tech coach to stand over 
their shoulder? 

In the end, the argument could be made 
that the low numbers for support indicate 
the correct amount was provided for an 
onsite event, since eventually the model 
calls for the skills to be learned either 
virtually or from a tutorial. Perhaps the low 
numbers were a good thing because if there 
had been too much hand holding, it would 
not have been replicable in a virtual 
environment. 

In conclusion, providing the right amount of 
support for live and virtual events can be a 
complex at a minimum. Deploying a virtual 
environment for training can require 
providing a replicable model for support that 
addresses a number of skill levels and 
learning styles. Numbers can be deceiving; 
high agreement with a support question 
could mean that the level of support cannot 
be replicated in a distributed environment. 

This delicate balance is a necessary one to 
achieve to avoid creating a barrier with the 
virtual technology rather than a tool. 
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